Docket Number: MS1-1754US Page: 1 of 13 
Inventor: Ma et al. 
Title: Robust Camera Motion Analysis of Home Video 



100 




I- 1° I , 



rwi 



LZZ] 



DOOODQD 







L_n a n 





CLIENT DEVICE 
104 



Application 118 



Video Analysis 
Module 116 



Video 
Storage 
112 



Docket Number: MS1-1754US Page: 2 of 13 
Inventor: Ma et al. 
Title: Robust Camera Motion Analysis of Home Video 




Docket Number: MS1-1754US Page: 3 of 13 
Inventor: Ma et al. 
Title: Robust Camera Motion Analysis of Home Video 



300 



VIDEO 
PROVIDER 
102(iri 



Video 
Storage 
108fn) 




CLIENT DEVICE 104 



PROCESSOR 302 



Application 118 



Video Analysis Module 116 



Motion 


Representation 


Module 306 




ITM310 J 






Motion 






Representation 






312 











Camera Motion Analysis 


Module 308 




Shot Boundary 






Detection 314 








Camera Shaking 






Detection 3Jh6 








Camera Motion 






Segmentation 318 








Camera Motion 






Characterization 






320 







MEMORY 
304 



Video 
Storage 
112 



m 



Video 1 



im 8 



'Pty. 3 



Docket Number: MS1-1754US Page: 4 of 13 
Inventor: Ma et al. 
Title: Robust Camera Motion Analysis of Home Video 




Docket Number: MS1-1754US Page: 5 of 13 
Inventor: Ma et al. 
Title: Robust Camera Motion Analysis of Home Video 



i L 

502 n V 




Docket Number: MS1-1754US Page: 6 of 13 
Inventor: Ma et al. 
Title: Robust Camera Motion Analysis of Home Video 




604 000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 
4 



2 

$ 0 
-2 









I 
















I J 












v 





606 -J 000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 

*2 



3 o 



608 



■ 




1 




1 


) I 


Jim 


S 




■ 


- 








i 






- 



^1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 
30 



£20 



T 



T 



T 



T 



610^1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 



R 

|h 

s 



- 






I 

1 


X 




— H 




- 




1 
1 
1 

- i ■ h 


- 


* 




h 





1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 



Docket Number: MS1-1754US Page: 7 of 13 
Inventor: Ma et al. 
Title: Robust Camera Motion Analysis of Home Video 




Docket Number: MS1-1754US Page: 8 of 13 
Inventor: Ma et al. 
Title: Robust Camera Motion Analysis of Home Video 




Docket Number: MS1-1754US Page: 9 of 13 
Inventor: Ma et al. 
Title: Robust Camera Motion Analysis of Home Video 



902 



ANALYZE VIDEO TO DETERMINE 
ONE OR MORE CAMERA MOTIONS 

THAT PERFORMED WHEN 
CAPTURING EACH OF THE FRAMES 
OF THE VIDEO 



904 



EXECUTE MEAN 
ABSOLUTE DIFFERENCE 
ALGORITHM 





ESTIMATE CAMERA 




906 — \ 


MOTION THROUGH 






ERROR 






MINIMIZATION 






i 


r 


908 — n 


DESCRIBE ONE OR MORE i 




CAMERA MOTIONS j 



Docket Number: MS1-1754US Page: 10 of 13 
Inventor: Ma et al. 
Title: Robust Camera Motion Analysis of Home Video 




1002 



1004 



DESCRIBE ONE OR MORE CAMERA 
MOTIONS THAT ARE PERFORMED 
WHEN CAPTURING SEQUENTIAL 
FRAMES OF A VIDEO 



I 



DETECT A SHOT BOUNDARY 
IN SEQUENTIAL FRAMES OF 
THE VIDEO 



1006 



DETECT AN ABRUPT 
TRANSITION FROM A SHARP 
PEAK IN A SLIDING WINDOW 

OF THE MAD CURVE 



1008 -n] SEGMENT THE VIDEO BASED 
ON THE SHOT BOUNDARY 



Docket Number: MS1-1754US Page: 11 of 13 
Inventor: Ma et al. 
Title: Robust Camera Motion Analysis of Home Video 



1102 



1104 



DESCRIBE ONE OR MORE CAMERA 
MOTIONS THAT ARE PERFORMED 
WHEN CAPTURING SEQUENTIAL 
FRAMES OF A VIDEO 



I 



DETECT CAMERA SHAKING IN 

THE VIDEO BASED ON THE 
DESCRIBED CAMERA MOTIONS 



COMPUTE MOTION ENERGY E AND 
VARIATION Var OF A SEGMENT OF 
THE VIDEO THAT IS DESCRIBED BY 
THE H, V, R DISPLACEMENT CURVES 



FIND A PROBABILISTIC 
1 1 08 - 1 DISTRIBUTION OF CAMERA 
SHAKING AND NON-SHAKING 
ALONG E AND Var AXES 



DETERMINE WHETHER THE 
SEGMENT INCLUDES CAMERA 

SHAKING FROM THE 
PROBABILISTIC DISTRIBUTION 



Docket Number: MS1-1754US Page: 12 of 13 
Inventor: Ma et al. 
Title: Robust Camera Motion Analysis of Home Video 



DESCRIBE ONE OR MORE CAMERA 
1202 -s| MOTIONS THAT WERE PERFORMED 
WHEN CAPTURING SEQUENTIAL 
FRAMES OF A VIDEO 



CHARACTERIZE EACH OF THE DESCRIBED ONE OR 

MORE CAMERA MOTIONS AS ONE OF STILL, 
CAMERA SHAKING, IRREGULAR CAMERA MOTION, 
AND ONE OR MORE REGULAR CAMERA MOTIONS 



1206 



1208 



PERFORM AN OPEN OPERATION TO 
REMOVE UNSTABLE MOTIONS FROM 
H, V, AND R DISPLACEMENT CURVES 



I 



PERFORM A CLOSED OPERATION 
TO REMOVE INTERRUPTED 
REGULAR MOTIONS 



1210 



I 



DETECT ONE OR MORE 
SHOT BOUNDARIES 



1212 



MERGE OVERLAP 
SEGMENTS 



I 



1214 



DETECT CAMERA 
SHAKING AND IRREGULAR 
CAMERA MOTION 



1216 



1 



I 



DETECT STILL 



12 



Docket Number: MS1-1754US Page: 13 of 13 
Inventor: Ma et al. 
Title: Robust Camera Motion Analysis of Home Video 



1300 



Remote 
Computing 
Device 




□□□□□ G 



1334 



Video Adapter 



Data Media 
Interfaces 



A 
V 



Network 
Adapter 



System Bus 



Operating 131 6 
System ^ 



Applicatiorv|3i8 
^Programs ^ 

Program 1320 
__ _Modules_ _^--^ 
Program 1322 
Data 



1324 



> 



1304 



1 11 




\7 



Processing 
Unit 

1342 



□□□□□ G 



Operating 
System 1316 



Application 
Programs 131 8 



Other Program 
Modules 1320 



Program 
Data 1322 



1312 



RAM 



BIOS 



1314 



1310 



ROM 



I/O Interfaces 




1340 



Printer 



Mouse 



Keyboard 



EL 



Other Device(s) 



